fix: recover tmux terminal streams after server restart - #597

[P1] Seeding bypasses the pyte screen, so a worker that was mid-turn at restart stays PROCESSING forever

seed_from_snapshot writes the snapshot into self._buffers and calls _detect_status — the raw path. It never touches self._screens. But _process_chunk:139-142 routes every provider with supports_screen_detection = True through the pyte screen whenever CAO_PYTE_STATUS is on, and that is the default (constants.py:228). Five providers opt in, including claude_code.

So after recovery the pyte screen is blank. The next real chunk off the re-armed FIFO is composited onto nothing, and TUI output is overwhelmingly cursor-addressed in-place repainting — which is the entire reason the screen path exists.

Reproduced at d9597de (pane mid-turn at restart; Ink then repaints row 2 in place to finish the turn):

A) PR #597  status_monitor.seed_from_snapshot()
   screen primed at recovery : False
   viewport after repaint    : ['✻ Crunched for 12s']
   seeded -> final status    : processing -> processing
   InboxService delivers?    : False

B) replay through _process_chunk (the existing primitive)
   screen primed at recovery : True
   viewport after repaint    : ['● Working on the task', '✻ Crunched for 12s',
                                '──────────', '❯', '──────────']
   seeded -> final status    : processing -> completed
   InboxService delivers?    : True

In (A) the truncated viewport yields UNKNOWN, which _apply_detection correctly suppresses because a known status is already latched — so the terminal is pinned at PROCESSING. An idle agent emits nothing further, so nothing ever re-triggers detection. The pending callback is never delivered, and only another restart clears it. That is precisely the failure this PR set out to fix; it just moved from "was idle at restart" to "was busy at restart", which is the more likely restart scenario.

The codebase already solved this. fifo_reader._rearm_stalled_pipe:686-688 recovers a dead forwarder by replaying the pane through the normal pipeline, and its docstring even documents the \r\n requirement:

replay = content.replace("\n", "\r\n")
bus.publish(f"terminal.{terminal_id}.output", {"data": replay})

That \r\n is not cosmetic — get_history joins with a bare \n and pyte defaults LNM off, so raw capture-pane output staircases:

bare \n:                        \r\n:
|● Working                      |● Working
|         ✻ Cultivating… (12s)  |✻ Cultivating… (12s)
|                     ────────  |────────

Please reuse that primitive instead of adding a parallel one. Keeping it synchronous (which you need, so the seed lands before the reconcile_orphaned_messages call at api/main.py:1102) is easy — _schedule_screen_detection detects on the rising edge inline when _bursting is False, which you already set:

        with self._lock:
            self._buffers[terminal_id] = ""
            self._screens.pop(terminal_id, None)
            self._bursting[terminal_id] = False
        # One code path for both detectors: primes the pyte screen, fills the
        # raw buffer, and picks the capability-correct detector.
        self._process_chunk(terminal_id, output.replace("\n", "\r\n"))
        return self.get_status(terminal_id)

And take the snapshot after _rearm_pipe() in terminal_service.py (Copilot's point at :240) so output produced during reattachment is included rather than lost to the old FIFO.

Whichever shape you land on, this needs a test that actually runs the real StatusMonitor — see my note on the test file.

[P2] Recovered idle terminals trip the cold-start watchdog and get permanently dropped from it

create_reader unconditionally sets _ever_delivered[terminal_id] = False and _registered_at[terminal_id] = now (fifo_reader.py:180-182). For a newly launched terminal that is a correct assumption — a starting CLI always emits something. For a recovered idle terminal it is wrong: the pane is quiescent by definition, so the FIFO will never deliver a byte.

The cold-start branch at fifo_reader.py:490-497 then matches on exactly that shape — not ever_delivered and past the grace window and content.strip() (a recovered pane is full of prior conversation) — and concludes "the forwarder never started, full stop." Note the self-ROAST comment at :498-508: the replay publishes straight to the bus and never flips _ever_delivered, so each attempt re-qualifies.

Reproduced at d9597de with the exact probe/rearm pair this function registers and an idle pane:

grace=3.0s  max_cold_start_attempts=5
after create_reader: _ever_delivered=False  enrolled=True
  tick 1: rearms=1  still_enrolled=True
  ...
  tick 5: rearms=5  still_enrolled=True
  tick 6: rearms=5  still_enrolled=False

ERROR pipe-pane forwarder for terminal t1 never started delivering after 5
      cold-start re-arm attempts — giving up and dropping it from the
      liveness watchdog

So every recovered idle supervisor — the exact population this PR targets — gets ~5 spurious stop_pipe_pane/pipe_pane cycles plus 5 full-pane replays in the first ~20s (PIPE_LIVENESS_CHECK_INTERVAL_S 4.0s), an ERROR log, and is then unenrolled from the issue #388 liveness watchdog for the life of the process. If its pipe genuinely stalls later, nothing repairs it — silently, since the give-up already logged and won't log again.

P2 rather than P1 because the immediate effect is loud and benign-to-helpful; what's lost is a safety net, quietly.

Recovery knows the pipe was just re-armed against a live pane, so the cold-start check is not applicable here. Give create_reader a way to say so and have recovery pass it — the ordinary divergence check still protects these terminals:

# fifo_reader.create_reader(..., assume_delivered: bool = False)
self._ever_delivered[terminal_id] = assume_delivered

# terminal_service, recovery path
fifo_manager.create_reader(
    terminal_id, pane_probe=_probe_pane, rearm=_rearm_pipe, assume_delivered=True
)

[P2] black fails here (CI red), and mocking status_monitor leaves the new logic completely untested

The Code Quality job fails on this file. Reproduced locally with uv run black --check:

-    list_terminals.return_value = [
-        {"id": "abc12345", "tmux_window": "tech_lead-a1b2"}
-    ]
+    list_terminals.return_value = [{"id": "abc12345", "tmux_window": "tech_lead-a1b2"}]

Separately — and this is why the P1 shipped unnoticed — the test patches terminal_service.status_monitor wholesale, so status_monitor.seed_from_snapshot.assert_called_once_with(...) only asserts that a MagicMock was called. seed_from_snapshot is referenced nowhere else in test/, so the real implementation is executed by zero tests. What the suite currently pins is the call wiring, not the recovery.

At minimum, add a test that drives the genuine StatusMonitor with a real provider:

def test_seed_primes_screen_so_a_mid_turn_pane_still_reaches_completed():
    sm, prov = StatusMonitor(), ClaudeCodeProvider("t1", "s", "w")
    busy = "\n".join(["● Working", "✻ Cultivating… (12s)", "─" * 60, "❯ ", "─" * 60])
    with patch("...status_monitor.provider_manager") as pm, \
         patch("...status_monitor.bus", MagicMock()):
        pm.get_provider.return_value = prov
        assert sm.seed_from_snapshot("t1", busy) == TerminalStatus.PROCESSING
        sm._process_chunk("t1", "\x1b[2;1H\x1b[2K✻ Crunched for 12s")  # in-place repaint
        sm._apply_detection("t1", sm._detect_screen("t1", prov))
    assert sm._last_status["t1"] == TerminalStatus.COMPLETED  # fails on d9597de

The failure paths in recover_persisted_terminal_output_streams are also untested — get_history raising (stale DB row), create_reader raising, and the list_sessions guard — and those except blocks are the load-bearing part of the "must never prevent server startup" contract in the docstring.